Create field filters using Splunk Web
Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.
READ THIS FIRST: Should you deploy field filters in your organization?
Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview
, mstats
, tstats
, typeahead
, and walklex
), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.
How to create field filters using Splunk Web
Field filters control visibility of data within events by redacting or obfuscating confidential information such as personal identifiable information (PII) and protected health information (PHI) data when users who are not exempt from the field filters run searches. This type of filter can remove fields from events returned by search results, or replace values of fields when those fields appear in the results. If you don't want unauthorized people to know that a field even exists, you can redact the field by deleting it from the search results. Alternatively, you can redact the field value by replacing it with a custom string, or you can replace the field value with a SHA-256 or SHA-512 (SHA-2 family) hash value that obfuscates the sensitive data. After you configure field filters, the settings then affect the results of searches run by all nonexempt users by filtering out specific indexed fields, including _raw fields.
Develop a plan for your field filters before you create them. See Plan for field filters in your organization.
Prerequisites
By default, to create, edit, or delete field filters, you must be a member of the admin or sc_admin role. To view field filters, you must be a member of the admin, sc_admin, or power user role. See Define roles on the Splunk platform with capabilities in Securing Splunk Platform.
Steps
This section describes how to use Splunk Web to create a new field filter by assigning a value to a field. To create and manage field filters, follow these steps:
- Select Settings and then Field filters from Users And Authentication.
- If this is the first time you are creating a field filter, you are prompted to acknowledge that there are certain restricted commands that users can't use by default on searches across indexes that have field filters. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.
- Select New field filter.
Now you have a choice of creating a field filter for an indexed or _raw field. The following steps describe how to:
- Create a field filter with limits for an indexed field.
- Create a field filter with limits for a _raw field using the sed
(s)
command. - Create a field filter with limits for a _raw field using the sed
(y)
command.
If your indexed fields are generated from _raw, keep confidential information in indexed and _raw fields safe by creating field filters for your indexed fields and a field filter for the _raw field from which the indexed fields are extracted. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.
Use the same replacement method for matching indexed and _raw field filters. For example, if a field filter for an indexed field uses a custom string XXX
, then XXX
should also be used to replace the field value for the matching _raw field.
Create a field filter with limits for an indexed field
Create a field filter for an indexed field that contains sensitive information that you want to redact or obfuscate, and limit your field filter to one or more hosts, sources, or source types.
- In the Field filter and limit page, enter a name and optional description for your new field filter.
- Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
- In the Limit type field, select a specific host, source, or source type, which is used to limit the field filter and optimize performance. Only one limit type is supported per field filter.
- If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
- In the Field type field, select Indexed to create a field filter for an indexed field.
- In the Indexed field field, enter the name of the field that you want to restrict with the field filter. For example, if you want to set a field filter on a sensitive field called
account_number
, the indexed field isaccount_number
.
You can't use wildcard characters to specify multiple fields in the fieldname because field filters treat wildcards like any other character. - Select Next.
- In the Replacement method page, select the method you want to use with this field filter to protect sensitive data.
The methods are described as follows:
Method Description Delete At search time, removes the specified indexed field and its value from results of searches to which this filter is applied. The field is still visible to exempt roles because the field is still present in the index.
Use this option if your sensitive data needs to be redacted with the highest form of privacy for legal or regulatory reasons.String Replaces the value of the field with the specified string wherever the field appears in results of searches to which this filter is applied.
Use this option if your sensitive data needs to be redacted with a high level of privacy while still indicating the existence of the field.SHA512 Computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-512 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference. SHA256 Computes and returns the secure hash of the value of the field based on the FIPS-compliant SHA-256 (SHA-2 family) hash function. This hash is then used to replace the value of the field wherever it appears in results of searches to which this filter is applied. See Cryptographic functions in the Splunk Cloud Platform Search Reference. - If you selected String as the replacement method, enter the string that you want to be used to replace the value of the field that this field filter will apply to.
- Select Next.
- In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter applies to all roles when they run searches to which this field filter applies.
- Select Next.
- Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.
Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.
After you've created a field filter for your indexed field, make sure that there is also a field filter protecting the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.
Use the same replacement method for matching indexed and _raw field filters. For example, if a field filter for an indexed field uses a custom string XXX
, then XXX
should also be used to replace the field value for the matching _raw field.
Create a field filter with limits for a _raw field using the sed (s)
command
The sed expression in a _raw field filter acts on searches to which the filter is applied. The sed expression replaces strings in search results that are matched by a regular expression (s).
If you create field filters for indexed fields that are generated from _raw, you should also create a field filter for the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.
Use the same replacement method for matching indexed and _raw field filters. For example, if a field filter for an indexed field uses a custom string XXX
, then XXX
should also be used to replace the field value for the matching _raw field.
The syntax for using the sed (s)
command to replace strings in search results that are matched by a regular expression (s) is s/<regex>/<replacement>/<modifiers>
. See About Splunk regular expressions in the Splunk Cloud Platform Knowledge Manager Manual.
When you create a _raw filter, make sure that the regular expression for that field filter doesn't conflict with an existing field filter.
Follow these steps to create a field filter for a _raw field using the sed (s)
command to replace strings in search results that are matched by a regular expression, and limit your field filter to one or more hosts, sources, or source types.
- In the Field filter and limit page, enter a name and optional description for your new field filter.
- Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
- In the Limit type field, optimize performance by limiting the field filter to a specific host, source, or source type. Only one limit type is supported per field filter.
- If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
- In the Field type field, select _raw to create a field filter for a _raw field.
- Select Next.
- In the Replacement method page, select s for the Sed command type that is used to specify a sed expression. The sed expression acts on searches to which this filter is applied and replaces strings in search results that are matched by a regular expression
(s)
. - In the Regex value match field, enter the regular expression that you want this field filter to apply to searches.
- In the Replacement field, enter a string that replaces the regular expression match. Use
\<n>
for back references, where<n>
is a single digit. - In the Command modifier field, select Global or Number. Other sed modifiers (also known as flags) for the
(s)
command are not supported. The modifiers are described as follows:Modifier Description Global Globally replace all matches. Number A number that specifies the position of the matching text to replace in the regular expression. - (Optional) If you selected Number, enter a number in the Modifier number field to specify the position of the matching text that should be replaced in the regular expression. For example, if you enter 2, the 2nd matching text in the string will be replaced when the field filter is applied to searches.
- Select Next.
- In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter will apply to all roles when they run searches to which this field filter applies.
- Select Next.
- Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.
Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.
Create a field filter with limits for a _raw field using the sed (y)
command
The sed expression in a _raw field filter acts on searches to which the filter is applied. The sed expression transliterates characters found in search results with corresponding characters provided by the sed expression (y).
If you create field filters for indexed fields that are generated from _raw, you should also create a field filter for the _raw field that generates the indexed fields. Otherwise, a user might be able to circumvent the indexed field filter by accessing the confidential data you are trying to protect that is contained in the _raw field.
Use the same replacement method for matching indexed and _raw field filters. For example, if a field filter for an indexed field uses a custom string XXX
, then XXX
should also be used to replace the field value for the matching _raw field.
The (y)
command syntax transliterates the source characters in search results with corresponding destination characters that you provide in the expression. For example, /: y/abc/def/
replaces a
with d
, b
with e
, and c
with f
. This expression changes the string aaabbc
to dddeef
. See About Splunk regular expressions in the Splunk Cloud Platform Knowledge Manager Manual.
When you create a _raw filter, make sure that the regular expression for that field filter doesn't conflict with an existing field filter.
Follow these steps to create a field filter for a _raw field using the sed (y)
command to transliterate characters found in search results with corresponding characters provided by the sed expression (y)
, and limit your field filter to one or more hosts, sources, or source types.
- In the Field filter and limit page, enter a name and optional description for your new field filter.
- Enter the name of the target indexes you want to search that contain the data you want to protect. Only events from these indexes are filtered by the field filter.
- In the Limit type field, optimize performance by limiting the field filter to a specific host, source, or source type. Only one limit type is supported per field filter.
- If you limited the field filter to a host, source, or source type, enter the name of one or more hosts, sources, or source types in Host limit, Source limit, or Source type limit.
- In the Field type field, select _raw to create a field filter for a _raw field.
- Select Next.
- In the Replacement method page, select y for the Sed command type that is used to specify a sed expression. The sed expression acts on searches to which this filter is applied and transliterates characters found in search results with corresponding characters provided by the sed expression
(y)
. - In the Source characters field, enter the source characters. Make sure that the Source characters and Replacement characters fields contain the same number of characters.
- In the Replacement characters field, enter the characters that you want to replace in the sed expression.
- Select Next.
- In the Exempt role from filter (optional) page, add any roles to the list of Roles without filter that you do not want the field filter to affect. If you do not select any roles to exempt from this field filter, the field filter will apply to all roles when they run searches to which this field filter applies.
- Select Next.
- Verify that your field filter is configured properly and then select Create. Be sure to test your new filter when you're done to make sure it is protecting your sensitive data in the way that you expect. Also, if certain trusted roles are exempt from the field filter, confirm that the roles still have access to the sensitive data in searches. It might take a few minutes for new field filters, or changes to existing field filters, to take effect as they're updated across the system.
Splunk software does not validate information you enter for your field filter. To correct an error in your field filter, make changes to your configuration before selecting Create, or edit the field filter after you create it.
Examples for creating field filters
The following examples show how you can use field filters to perform operations on fields in searches run by nonexempt roles.
1. Delete a field from search results
At your organization, the indexed field patient_name
is sensitive for security reasons. You have a role named managers, and you want people who belong to the managers role to be able to access the patient_name
field in their search results. But, you don't want users with other roles to be able to see the patient_name
field.
If you want to redact the field from the results of searches for everyone but people holding the managers role, create an indexed field filter for the patient_name
field with the replacement method set to Delete and exempt the managers role. This removes the patient_name
field from the results of searches, so people who aren't managers won't even know the field exists. The managers role that you exempted can still see the field in searches to which the field filter applies.
Removing a field and its value is the most secure method of protection because someone with malicious intentions can't deduce anything about the data if there is no trace of it in the search results.
2. Redact a field value
If you want to hide your field values, but not remove them from the search results entirely, you can set your field filter to replace the field value with a custom string. For example, if you want all users except for people who hold the staff role to see search results such as patient_name = XXXX
, set a field filter to a custom string and exempt the staff role from the field filter.
Now when anyone who doesn't hold the staff role searches on events that include the patient_name
field, the string XXX
will appear in the results in place of the actual confidential information about the patient. Users who hold the staff role can still see the confidential data as usual.
3. Obfuscate a field value
You can also set your field filter to replace the field value with a hash value. For example, if you want all users to see a hash value instead of a patient's account number in search results, create a field filter for the patient_name
field and set the replacement method to SHA256.
Now when someone searches on events that include the patient_account
field, a hash value appears in the results in place of the actual confidential information about the patient.
4. Configure an indexed field and corresponding _raw field
To protect sensitive data in both indexed and _raw fields, create a field filter for an indexed field and a _raw field filter for the undefined field that is used to generate the indexed field. Follow these steps when creating a new field filter for the indexed field.
- Specify the name of this field filter, for example,
auth_filter
. - Add a description for the field filter, such as
Provide a field filter on authentication logs.
- Specify the name of the target index, such as
web_access_index
. - Specify the limit type, such as
source type
. - Specify the name of source type limit, for example,
corp_auth_log
. - Select the
indexed
field type. - Specify the name of the indexed field, such as
password
. - Specify the action you want the field filter to perform, such as replace the field value with a string such as
Redacted
. - If a highly privileged role is exempt from this field filter, such as the "exempt" role, add the role to the list of roles without the filter.
- The Field filter summary page looks like this:
Create a field filter on the _raw field, so that _raw is treated as a simple indexed field and its value is redacted or hidden. Follow these steps to create a _raw field filter that corresponds to the indexed field filter you just created.
- Specify the name of the field filter, for example,
auth_raw_filter
. - Add a description for the field filter, such as
Provide a _raw field filter on authentication logs
. - Specify the name of the target index, such as
web_access_index
. - Specify the limit type, such as
source type
. - Specify the name of the source type limit, for example,
corp_auth_log
. - Select the
_raw
field type. You don't need to specify the name of the _raw field because it is always called _raw. - Since a _raw field filter has an enhanced action that uses the Sed command to replace a partial string of the _raw field, use the
s
ory
option. For example, use thes
option to replacepassword=<my_pass_word>
withpassword=REDACTED,
- The field filter summary page looks like this:
Now the password
field is protected with both an indexed field filter and a _raw field filter.
5. Create two field filters for an indexed and a _raw field
To make sure that both the indexed and source data for a field are protected, say you create the following two field filters, which mask the host=
and user=
fields for all users:
- A field filter for an indexed field called
host
that is set to replace the field and its value with the stringunknown host
. - A field filter for a _raw field called
user
that uses the sed (s) command to replace the string that is matched by the regular expressions/user=[^ ]+/REMOVED-USER/
.
Then, the host name and username are redacted when a user runs the following search:
index="main"
The search results might look like this:
Time | Event |
---|---|
2023-01-31
15:01:58.679 |
[timestamp=01-31-2023 15:01:58.679, REMOVED-USER action=search, info=granted REST: /search/ast]
host = unknown host |
2023-01-31
15:01:58.678 |
[timestamp=01-31-2023 15:01:58.678, REMOVED-USER action=search, info=granted REST: /search/timeparser]
host = unknown host |
2023-01-31
15:01:58.677 |
[timestamp=01-31-2023 15:01:58.677, REMOVED-USER action=edit_win_eventlogs, info=denied ]
host = unknown host |
2023-01-31
15:01:58.676 |
[timestamp=01-31-2023 15:01:58.676, REMOVED-USER action=edit_win_regmon, info=denied ]
host = unknown host |
6. Hide an indexed or _raw field from search results
To hide an indexed or _raw field from search results, set the field filter replacement method for your field filter as follows:
- In an indexed field filter, set to
Delete
. - In a _raw field filter, set
sed("s/.*//")
.
7. Create multiple field filters for the same field with different role exemptions
Say you have two roles called analyst and user that need different access to social security number data. For analysts who need to run statistics on social security numbers, the SSN field must be hashed. For users, the value of the SSN field must be redacted.
To set up field filters and roles according to these requirements, you might do the following:
- Create a field filter called
ssn_hash
that sets the value of the indexed SSN field to SHA-256. - Exempt users from the
ssn_hash
field filter, so users can still see the redacted SSN field. - Create a field filter called
ssn_redact
that redacts the value of the indexed SSN field by replacing it with the stringREDACTED
. - Exempt the analyst role from the
ssn_redact
field filter, so analysts can still see the data.
Now the SSN field is protected in the right way for each role and each role executes only one field filter.
8. Test a field filter that replaces a _raw field with a static value
This example shows you how to create and test a field filter that replaces the value for a _raw field called healthId
with a Universally Unique Identifier (UUID) of 00000000-0000-0000-0000-000000000000
.
- In Splunk Web, select Settings and then Field filters from Users And Authentication.
- Select New field filter.
- In the Field filter and limit page, enter the name
healthId_ff
for your new field filter. - Select
_raw
as the field type and then select Next. - In the Replacement method page, select s for the Sed command type
- In the Regex value match field, enter the regular expression
"healthId": "([^"]+)"
that you want this field filter to apply to searches. - In the Replacement field, enter the string
"healthId":"00000000-0000-0000-0000-000000000000"
that replaces the regular expression match. - In the Command modifier field, select Global. The Replacement method page for the field filter looks like this:
- Select Next, and Next, and then Create.
Then, run the following search to test that the field filter is correctly replacing the value of the healthId
field:
index="main" | head 1 | table healthId _raw
The search results look like this:
healthId | _raw |
---|---|
00000000- 0000-0000- |
{"user": "user-842", "eventTime": "2024-08-06T14:52:20", "healthId": "00000000-0000-0000-0000-000000000000", "field2": "7d92ee29-fd9b-4db3-bf5c-38dbbf821921", "field3": "2c85eb0d-4a1d-4c52-9a59-226b93d552e6"} |
The replacement value for healthId
is displayed in the healthId
field, and the _raw
field shows that healthId
has been replaced with 00000000-0000-0000-0000-000000000000
in all places where it appears in the results.
Examples for field filter searches
1. Hash a field value and get the count
An important consideration to performing hash functions on field values using SHA encryption is that field filters preserve the uniqueness of the field. That means that even if you encrypt identifying information, such as user ID numbers, you can still get a count of the number of user IDs without exposing the value. For example, say you create a field filter that sets the value of the host
field to a SHA-256 hash.
Then, you run the following search:
index=main | stats count by host
Your search results might look like this:
host | count |
---|---|
C1fe560f903b48c9681269132a783a9333414bbf7ac26bf41cebbaf79b6d | 8223 |
Notice that even though there is only one hash value, the count of user IDs is 8223.
2. Redact PII and PHI data in search results
You can use field filters to redact sensitive fields such as Personal identifiable information (PII) and Protected Health Information (PHI) from searches of all of your event data. For example, perhaps your search results include the name and phone number for a patient named Alex Martin, which looks like this:
Time | Event |
---|---|
2/17/23
3:57:04.597 PM |
[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search patient_name=AlexMartin phone=+1-555-555-0199', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"] |
If you want to replace patient_name=AlexMartin
and phone=+1-555-555-0199
with custom strings, you would create two _raw field filters for users who don't have authorization to see that sensitive information. One _raw field filter might be called staffName and is set to s/patient_name=[^ ']+/REMOVED-NAME/
. The other _raw field filter might be called staffPhone and is set to s/phone=[^ ']+/REMOVED-PHONE/.
Then, when a person who holds a role that is affected by these field filters run the following search:
index=main
The person now sees only search results for the patient with REMOVED_NAME REMOVED_PHONE
, instead of patient_name=AlexMartin phone=+1-555-555-0199
, which looks like this:
Time | Event |
---|---|
2/17/23
3:57:04.597 PM |
[timestamp=02-17-2023 15:57:04.597, user=test_api, action=search, info=granted , search_id='1645142224.7', search='search REMOVED-NAME REMOVED-PHONE', autojoin='1', buckets=300, ttl=600, max_count=500000, maxtime=8640000, enable_lookups='1', extra_fields=, apiStartTime='Thu Feb 17 15:42:04 2023', apiEndTime='Thu Feb 17 15:57:04 2023', apiIndexStartTime='ZERO_TIME', apiIndexEndTime='ZERO_TIME', savedsearch_name="", is_proxied=false, app="search", provenance="UI:Search", mode="historical"] |
The confidential data is no longer visible to all nonexempt roles that are affected by the field filters because the patient name and phone number have been replaced with custom strings.
3. Search for a redacted field
Searching for an index-time field might not return any results because the value of the filtered field is redacted at search time. For example, say you have an index-time field called password
and you run the following search:
index=web_access_index sourcetype::corp_auth_log password=REDACTED
The search doesn't return any events because password=REDACTED
is not indexed.
To search for the redacted field, run the following search:
index=web_access_index sourcetype::corp_auth_log
| table _time, password, _raw
| where password=REDACTED
See also
Turn on Splunk platform field filters | Optimize field filter performance using Splunk Web |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.2.2406 (latest FedRAMP release)
Feedback submitted, thanks!